Counting-by-Detection with Three-dimensional Fully Convolutional Networks

The project goal is to count and detect instances in an image using deep learning. We target the regime where object detectors need to work reliably in scenarios with crowding, overlapping or instances with a small size. To estimate the objects, two approaches have been used: counting-by-detections and counting- by-density-estimation. We concluded that objects could be rated simply, accurately and efficiently using a counting system and a classification model. To support this statement, we compare different experi- ments using different FCNs. Finally, we propose a quantitative and qualitative method, to evaluate the network. The results are demonstrated on a variety of visual material, including graphs, microscopy and density images.El objetivo del proyecto es contar y detectar instancias en una imagen mediante el aprendizaje profundo. Nuestro objetivo es un entorno en el que los detectores de objetos deben trabajar de manera correcta en escenarios abarrotados, con superposiciones o instancias con un tamaño pequeño. Para estimar los objetos, se han utilizado dos enfoques: recuento por detección y recuento por estimación de densidad. Concluimos que los objetos pueden clasificarse de manera simple, precisa y eficiente uti- lizando un sistema de conteo y un modelo de clasificación. Para apoyar esta afirmación, comparamos diferentes experimentos usando diferentes FCN. Finalmente, proponemos un método cuantitativo y cualitativo para evaluar la red. Los resul- tados se demuestran en una variedad de material visual, incluidos gráficos, microscopía e imágenes de densidad.L'objectiu del projecte és explicar i detectar instàncies en una imatge mitjançant l'aprenentatge profund. El nostre objectiu és un entorn en el qual els detectors d'objectes han de treballar de manera correcta en escenaris abarrotats, amb superposicions o instàncies amb una mida petita. Per estimar els objectes, s'han utilitzat dos enfocaments: recompte per detecció i recompte per estimació de densitat. Concloem que els objectes poden classificar-se de manera simple, precisa i eficient utilitzant un sistema de detecció i un model de classificació. Per donar suport a aquesta afirmació, comparem diferents experiments utilitzant diferents FCN. Finalment, proposem un mètode quantitatiu i qualitatiu per avaluar la xarxa. Els resultats es demostren en una varietat de material visual, incloent gràfics, microscòpia i imatges de densitat

RNN-CNN hybrid model to predict C-ATC CAPACITY regulations for en-route traffic

Meeting the demand with the available airspace capacity is one of the most challenging problems faced by Air Traffic Management. Nowadays, this collaborative Demand–Capacity Balancing process often ends up enforcing Air Traffic Flow Management regulations when capacity cannot be adjusted. This process to decide if a regulation is needed is time consuming and relies heavily on human knowledge. This article studies three different Air Traffic Management frameworks aiming to improve the cost-efficiency for Flow Manager Positions and Network Manager operators when facing the detection of regulations. For this purpose, two already tested Deep Learning models are combined, creating different hybrid models. A Recurrent Neural Network is used to process scalar variables to extract the overall airspace characteristics, and a Convolutional Neural Network is used to process artificial images exhibiting the specific airspace configuration. The models are validated using historical data from two of the most regulated European regions, resulting in a novel framework that could be used across Air Traffic Control centers. For the best hybrid model, using a cascade architecture, an average accuracy of 88.45% is obtained, with an average recall of 92.16%, and an average precision of 86.85%, across different traffic volumes. Moreover, two different techniques for model explainability are used to provide a theoretical understanding of its behavior and understand the reasons behind the predictionsThis work was funded EUROCONTROL under Ph.D. Research Contract No. 18-220569-C2 and by the Ministry of Economy, Industry, and Competitiveness of Spain under GrantNumber PID2020-116377RB-C21. This project has also received funding from the SESAR Joint Undertaking under the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 783287.Peer ReviewedPostprint (published version

Image-based multi-agent reinforcement learning for demand–capacity balancing

Air traffic flow management (ATFM) is of crucial importance to the European Air Traffic Control System due to two factors: first, the impact of ATFM, including safety implications on ATC operations; second, the possible consequences of ATFM measures on both airports and airlines operations. Thus, the central flow management unit continually seeks to improve traffic flow management to reduce delays and congestion. In this work, we investigated the use of reinforcement learning (RL) methods to compute policies to solve demand–capacity imbalances (a.k.a. congestion) during the pre-tactical phase. To address cases where the expected demands exceed the airspace sector capacity, we considered agents representing flights who have to decide on ground delays jointly. To overcome scalability issues, we propose using raw pixel images as input, which can represent an arbitrary number of agents without changing the system’s architecture. This article compares deep Q-learning and deep deterministic policy gradient algorithms with different configurations. Experimental results, using real-world data for training and validation, confirm the effectiveness of our approach to resolving demand–capacity balancing problems, showing the robustness of the RL approach presented in this article.This work was funded by EUROCONTROL under Ph.D. Research contract no. 18-220569- C2 and by the Ministry of Economy, Industry, and Competitiveness of Spain under grant number PID2020-116377RB-C21.Peer ReviewedPostprint (published version

Dispatcher3 – Machine learning for efficient flight planning: approach and challenges for data-driven prototypes in air transport

Machine learning techniques to support decisionmaking processes are in trend. These are particularly relevant in the context of flight management where large datasets of planned and realised operations are available. Current operations experience discrepancies between planned and executed flight plan, these might be due to external factors (e.g. weather, congestion) and might lead to sub-optimal decisions (e.g. recovering delay (burning extra fuel) when no holding is expected at arrival and therefore it was no needed). Dispatcher3 produces a set of machine learning models to support flight crew pre-departure, with estimations on expected holding at arrival, runway in use and fuel usage, and the airline’s duty manager on pre-tactical actions, with models trained with a larger look ahead time for ATFM and reactionary delay estimations. This paper describes the prototype architecture and approach of Dispatcher3 with particular focus on the challenges faced by this type of data-driven machine learning models in the field of air transport ranging: from technical aspects such as data leakage to operational requirements such as the consideration and estimation of uncertainty. These considerations should be relevant for projects which try to use machine learning in the field of aviation in general.This work is performed as part of Dispatcher3 innovation action which has received funding from the Clean Sky 2 Joint Undertaking (JU) under grant agreements No 886461. The Topic Manager is Thales AVS France SAS. The JU receives support from the European Union’s Horizon 2020 research and innovation programme and the Clean Sky 2 JU members other than the Union. The opinions expressed herein reflect the authors’ views only. Under no circumstances shall the Clean Sky 2 Joint Undertaking be responsible for any use that may be made of the information contained herein.Postprint (published version

Pre-tactical prediction of atfm delay for individual flights

Airlines develop an operation plan, during the day prior to operations (D-1), to identify potential network issues and prepare potential pre-tactical preventing measures such as aircraft tail swapping or crew reassignment to be applied on D0. Flights might experience discrepancies between their plan and execution due to many different factors, and in particular demand-capacity imbalances in the network leading to Air Traffic Flow Management (ATFM) regulations. Dispatcher3, a Clean Sky 2 innovation action, focuses on the use of machine learning techniques to support the airlines processes prior departure: dispatching, understood as the broad flight planning from the day prior to operations to the flight plan definition and selection, and advisories to pilot. This paper focuses on the estimation of ATFM delay for individual flights during the pre-tactical phase (D-1), which could help airspace users apply mitigation actions. Four machine learning models are developed to produce individual independent estimations with different level of granularity. The first two are binary classifier models that provide information on the probability of a given flight being affected by ATFM delay, and the reason for this delay (airport or airspace congestion). These models reported an accuracy between 75% and 88%. The later two models estimate the impact of the delay (amount of delay assigned to the flight if regulated), with a Mean Absolute Error close to 9.35 minutes.This work has been performed as part of Dispatcher3 innovation action which has received funding from the Clean Sky 2 Joint Undertaking (JU) under grant agreements No 886461. The JU receives support from the European Union’s Horizon 2020 research and innovation programme and the Clean Sky 2 JU members other than the Union. The opinions expressed herein reflect the authors’ views only. Under no circumstances shall the Clean Sky 2 Joint Undertaking be responsible for any use that may be made of the information contained herein.Peer ReviewedPostprint (published version

Counting-by-Detection with Three-dimensional Fully Convolutional Networks

The project goal is to count and detect instances in an image using deep learning. We target the regime where object detectors need to work reliably in scenarios with crowding, overlapping or instances with a small size. To estimate the objects, two approaches have been used: counting-by-detections and counting- by-density-estimation. We concluded that objects could be rated simply, accurately and efficiently using a counting system and a classification model. To support this statement, we compare different experi- ments using different FCNs. Finally, we propose a quantitative and qualitative method, to evaluate the network. The results are demonstrated on a variety of visual material, including graphs, microscopy and density images.El objetivo del proyecto es contar y detectar instancias en una imagen mediante el aprendizaje profundo. Nuestro objetivo es un entorno en el que los detectores de objetos deben trabajar de manera correcta en escenarios abarrotados, con superposiciones o instancias con un tamaño pequeño. Para estimar los objetos, se han utilizado dos enfoques: recuento por detección y recuento por estimación de densidad. Concluimos que los objetos pueden clasificarse de manera simple, precisa y eficiente uti- lizando un sistema de conteo y un modelo de clasificación. Para apoyar esta afirmación, comparamos diferentes experimentos usando diferentes FCN. Finalmente, proponemos un método cuantitativo y cualitativo para evaluar la red. Los resul- tados se demuestran en una variedad de material visual, incluidos gráficos, microscopía e imágenes de densidad.L'objectiu del projecte és explicar i detectar instàncies en una imatge mitjançant l'aprenentatge profund. El nostre objectiu és un entorn en el qual els detectors d'objectes han de treballar de manera correcta en escenaris abarrotats, amb superposicions o instàncies amb una mida petita. Per estimar els objectes, s'han utilitzat dos enfocaments: recompte per detecció i recompte per estimació de densitat. Concloem que els objectes poden classificar-se de manera simple, precisa i eficient utilitzant un sistema de detecció i un model de classificació. Per donar suport a aquesta afirmació, comparem diferents experiments utilitzant diferents FCN. Finalment, proposem un mètode quantitatiu i qualitatiu per avaluar la xarxa. Els resultats es demostren en una varietat de material visual, incloent gràfics, microscòpia i imatges de densitat

A novel methodology to predict regulations using deep learning

The current air traffic control system tries to allocate as many flights as possible in a scenario that is expected to be time-efficient, cost-efficient, and safe. To guaranty these safety conditions, it is performed a cyclic process known as Demand-Capacity Balancing. During this process, a specialized Air Traffic Controller analyses the situations where the demand is over the capacity to identify the required corrective actions. These corrective actions are mostly in the form of regulations, and they are necessary to avoid overload during the day of operation. The task of declaring a regulation is complicated, very time-consuming, and based on the Air Traffic Controller's experience. A massive amount of information must be considered simultaneously, together with a risk maturation process because of the uncertainty and granularity in the information. This paper proposes and evaluates two Deep Learning models able to mimic the current procedure's behavior, and therefore, helping the specialized Air Traffic Controller to automatically detect the imbalances that will require regulation. Both models, one based on Convolutional Neural Networks, and the second one based on Recurrent Neural Networks, have demonstrated the potential to predict regulations, with an accuracy of 81.45% and 80.73% respectively over the entire MUAC region in 30-minute intervals. This accuracy can be increased by up to 91% by developing specialized models for each airspace sector. Additionally, we performed an in-depth analysis of the most relevant features using SHapley Additive exPlanations.This work was funded EUROCONTROL under Ph.D. Research Contract No. 18-220569-C2 and by the Ministry of Economy, Industry, and Competitiveness of Spain under GrantNumber TRA2016-77012-R.Postprint (author's final draft

Predict ATFCM weather regulations using a time-distributed recurrent neural network

In recent years, prior to COVID-19, capacity shortfalls in airspace and airports inevitably caused an increase in aircraft delays. Therefore, when it returns to normal conditions, the airspace will exhibit the same capacity limits, even under normal weather conditions. To ensure that air traffic remains safe, reliable, and efficient in adverse weather conditions, planning and coordination activities through a Collaborative Decision Making process are required to deliver the most effective Air Traffic Flow and Capacity Management services to Air Traffic Control and Aircraft Operators. Nowadays, this task is based on air traffic controllers’ experience and historical data. That means that the Flow Manager Positions and the Network Manager operators have to process a huge amount of information, and the detection of future overloads is based on past experiences. Moreover, due to the inherent uncertainty of weather information, a reliable decision support framework is required to handle these situations as efficiently as possible. We propose a Deep Learning model able to extract the relationship between both the historical data and the implemented actions, accurately identifying the intervals of time that must be regulated. The proposed model achieves an accuracy between 80% and 90% across six traffic volumes belonging to both the MUAC and REIMS regions, a recall higher than 85%, and an F1-score higher than 0.8 in all the cases. Furthermore, the confidence-level analysis shows a really high activation when making a prediction. Finally, the SHapley Additive exPlanations method is applied to identify the most relevant input features.This work was funded EUROCONTROL under Ph.D. Research Contract No. 18-220569-C2 and by the Ministry of Economy, Industry, and Competitiveness of Spain under GrantNumber TRA2016-77012-R.Peer ReviewedPostprint (published version

Dispatcher3 - D5.2: Verification and validation report

The deliverable provides the results from the verification and validation activities within Dispatcher3 project. The document reviews the internal and external validation activities that were carried out during the course of Dispatcher3 project according to the plan defined in D5.1. Dispatcher3 is organised in three layers: data acquisition and preparation, predictive layer, with machine learning models, and prospective models, with the integration of the individual machine learning models in an interactive Advice Generator and an estimator of rotation/reactionary delay. This deliverable presents the verification and validation activities performed on these three components. For the data acquisition and preparation layer the data-pipelines, including the transformation verification and validation activities are described. In the predictive layer both the models developed for the first release and their evolution for the final prototype are described and presented. Finally, for the prospective layer, the interactive interface with its functional requirements is presented and verified, while the reactionary delay model is described, and different scenarios evaluated for its validation. The deliverable also describes the different internal and external activities and meetings, workshops and dedicated online site visits that have been performed during the duration of the project. Finally, the document assesses the verification of the high-level system-wide requirements identified at the beginning of the project in D1.1 – Technical resources and problem definition, and the research questions identified in the Verification and validation plan (D5.1).This deliverable is part of a project that has received funding from the Clean Sky Joint Undertaking under grant agreement No 886461 under European Union’s Horizon 2020 research and innovation programme.Peer ReviewedPostprint (published version